We introduce a method to measure the alignment between public will and language model (LM) behavior that can be applied to fine-tuning, online oversight, and pre-release safety checks. Our `chain of alignment' (CoA) approach produces a rule based reward (RBR) by creating model behavior $\textit{rules}$ aligned to normative $\textit{objectives}$ aligned to $\textit{public will}$. This factoring enables a nonexpert public to directly specify their will through the normative objectives, while expert intelligence is used to figure out rules entailing model behavior that best achieves those objectives. We validate our approach by applying it across three different domains of LM prompts related to mental health. We demonstrate a public input process built on collective dialogues and bridging-based ranking that reliably produces normative objectives supported by at least $96\% \pm 2\%$ of the US public. We then show that rules developed by mental health experts to achieve those objectives enable a RBR that evaluates an LM response's alignment with the objectives similarly to human experts (Pearson's $r=0.841$, $AUC=0.964$). By measuring alignment with objectives that have near unanimous public support, these CoA RBRs provide an approximate measure of alignment between LM behavior and public will.

Key Words: Mental health chatbots, large language models, clinical safety, evaluation metrics, automated assessment Word Count: 3,686 ABSTRACT Objective: This study aims to develop and validate an evaluation framework to ensure the safety and reliability of mental health chatbots, which are increasingly popular due to their accessibility, human-like interactions, and context-aware support. Materials and Methods: We created an evaluation framework with 100 benchmark questions and ideal responses, and five guideline questions for chatbot responses. This framework, validated by mental health experts, was tested on a GPT-3.5-turbo-based Automated evaluation methods explored included large language model (LLM)-based scoring, an agentic approach using real-time data, and embedding models to compare chatbot responses against ground truth standards. The agentic method, dynamically accessing reliable information, demonstrated the best alignment with human assessments. Discussion: Our findings emphasize the need for comprehensive, expert-tailored safety evaluation metrics for mental health chatbots. While LLMs have significant potential, careful implementation is necessary to mitigate risks. The superior performance of the agentic approach underscores the importance of real-time data access in enhancing chatbot reliability. Future work should extend evaluations to accuracy, bias, empathy, and privacy to ensure holistic assessment and responsible integration into healthcare. Standardized evaluations will build trust among users and professionals, facilitating broader adoption and improved mental health support through technology.

A proven therapeutic technique to overcome negative thoughts is to replace them with a more hopeful "reframed thought." Although therapy can help people practice and learn this Cognitive Reframing of Negative Thoughts, clinician shortages and mental health stigma commonly limit people's access to therapy. In this paper, we conduct a human-centered study of how language models may assist people in reframing negative thoughts. Based on psychology literature, we define a framework of seven linguistic attributes that can be used to reframe a thought. We develop automated metrics to measure these attributes and validate them with expert judgements from mental health practitioners. We collect a dataset of 600 situations, thoughts and reframes from practitioners and use it to train a retrieval-enhanced in-context learning model that effectively generates reframed thoughts and controls their linguistic attributes. To investigate what constitutes a "high-quality" reframe, we conduct an IRB-approved randomized field study on a large mental health website with over 2,000 participants. Amongst other findings, we show that people prefer highly empathic or specific reframes, as opposed to reframes that are overly positive. Our findings provide key implications for the use of LMs to assist people in overcoming negative thoughts.

Before responding to Lum's typed messages, the program adds the preface: "As an AI language model, I am not a licensed therapist, and I am unable to provide therapy or diagnose any conditions. However, I am here to listen and help in any way I can." For Lum, who uses ChapGPT to pour out her feelings and not much more -- that's exactly what she's looking for. "I often feel better after using online tools for therapy, and it certainly aids my mental and emotional health," Lum told BuzzFeed News. "I enjoy being able to unload my thoughts on ChatGPT, and would consider this an improvement from journaling because I am able to receive feedback on my thoughts and situation."

A social media platform designed for America's military community is now equipped with a custom machine learning model that insiders say can rapidly review public posts and pinpoint those that show signs and risks of potential self-harm. With support from the Veterans Affairs Department and Harvard University's Nock Lab, Amazon Web Services linked up with the existing RallyPoint military social media platform to target the production of a technological solution that can speedily surface sensitive public posts and boost online suicide intervention. "Historically, the heavy lifting of mental health support on RallyPoint has been shouldered by RallyPoint members stepping up to help each other when they come across people sharing their challenges on our site," RallyPoint CEO Dave Gowel recently told Nextgov. "Now, through our work with the VA, AWS and mental health experts from Harvard, we are more proactive in reinforcing our members' good work by offering helpful resources when we are alerted about public posts showing signs of risk." Launched in 2012, RallyPoint enables nearly 2 million service members, veterans, and their families to connect, share stories and information, ask questions and ultimately chat on topics that accompany military and veteran life.

"Talk therapy" is often used by psychotherapists to help patients overcome depression or anxiety through conversation. A research team at Massachusetts Institute of Technology is using deep learning to uncover what might be called "talk diagnosis" -- detecting signs of depression by analyzing a patient's speech. The research could lead to effective, and inexpensive, diagnosis of serious mental health issues. An estimated one in 15 adults in the U.S. reports having a bout of major depression in any given year, according to the National Institute of Mental Health. The condition can lead to serious disruptions in a person's life, yet our understanding of it remains limited.

At least one in every five adults are affected by mental difficulties in the United States. On a global level, the number is much bigger, with alone in Europe, almost 83 million people are suffering from mental health issues. However, where there is a problem, technology always has a solution. How about productively using Artificial Intelligence (AI) to solve this issue? This might sound surprising to many but, AI is already being used to treat mental health issues or even examine your voice for any evidence of depression.